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Abstract 

The new perspectives on Bayesian model criticisms presented in 



Ratmann et al. ( 2009 1 are challenging standard approaches to Bayesian 
model choice. We discuss here some issues arising from the approach, 
including prior influence, model assessment and criticism, and the 
meaning of error. 
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In Ratmann et al. 


(2009 


1, the perception of the approximation error in 


the ABC algorithm (Pritchard et al. 


1999 


Beaumont et al. 


2002 


Marjoram 



et al. 20031 is radically modified, moving from a computational parameter 
that is calibrated by the user when balancing precision and computing time 
into a genuine parameter e about which inferences can be made in the same 



manner as for the original parameter 6. As stressed in Section S2 of Ratmann 



et al. (20091, this is indeed a change of perception rather than a modifica- 
tion of the ABC method in that the target in 6 remains the same. (This 
should not be construed as a criticism in that the unification of most ABC 
representations proposed in Section 2 is immensely valuable.) Although the 
derivation of the distribution ^ 0i 6»(e) is somewhat convoluted in Section SI, 
we note here that it is simply the distribution of the error p(§(x), §(xo)) 
when x ~ f(x\6), i.e. a projection of f(x\6) in probabilistic terms. 

Example — For a Poisson xo ~ "P(&) model, a natural divergence is the 
difference e = x — xq which is distributed as a translated Poisson V(9) — xo 
when conditional on xq and which is marginaly distributed as the difference 
of two iid V{9) variables. Since e thus is an integer valued variable, the 
supplementary prior 7r e should reflect this feature. A natural solution is 



vr e (fc) oc 1/(1 + k 2 



1 



since the series 1 /k 2 is converging, even though using a proper prior 7T e 



does not appear to be a necessary condition in Ratmann et al. (2009) 



The change of perception in Ratmann et al. (2009) is based on the un- 



derlying assumption that the data is informative about the error term e, 
which is not necessarily the shown by the previous and following 

examples. 



f(x — 9), if we take e 



xq, the 



Example — For a location family, xq 
posterior distribution of e is 

7r t (e\x ) oc J f(e + x - 9)Tr e (6)ir t (e)d9ir(e) 

and therefore a mostly flat prior ttq(6) with a large support produces a 
posterior 7r e (e|a;o) identical to ir t (e) for most values of xq. Conversely, a 
highly concentrated prior 7r e (e) hardly modifies the posterior tt(9\xq). M 

Example — For the binomial model xq ~ B(n, 9), assuming a uniform prior 
9 ~ ^/(0, 1), we can consider e = x — xo, in which case e is supported on 
{— re, . . . , n}. If we use a uniform prior on e as well, 



7r e (e\x ) 



oc 



oc 



n 

e + x 
n 

e + x 
1/(1 + 2 



e + x )\(n 



9) 1 



-■fo 



d0I 



so)!, 



(n+1)! 

..,«}(*) 



and therefore the (Bayesian) model brings no information about e. 

Obviously, this example is not directly incriminating against the method 



of Ratmann et al. (2009), in that it only considers a single statistic, instead 



of several as in Ratmann et al. ( 2009 ) (which distinguishes this paper from 
the remainder of the literature, where e is a single number). 



1 Bayesian model assessment 

The paper chooses to assess the validity of the model based on the marginal 
likelihood m(x) instead of the predictive p(x\xo). While this has the advan- 
tage of "using the data once" , it suffers from a strong impact of the prior 
modelling and of not conditioning on the observed data Xo- A more appro- 
priate (if still ad-hoc) procedure is to relate the observed statistics S(xq) 



with statistics simulated from p(x\xo), as in, e.g., Verdinelli and Wasser- 



man (1998). It may be argued that checking the prior adequacy is a good 
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thing, but having no way to distinguish between prior and sampling model 
inadequacy is a difficulty, as seen in the Poisson example. 

Example — For the location family, x$ ~ f(x — 6), the joint posterior dis- 
tribution of (#, e) is 

/(e + x - 9)ir(6)Tr(e) , 

and therefore the difference (e — 9) is not identifiable from the data, solely 
from the prior(s). < 

Note that, from an ABC perspective, using p(x\xo) instead of m{x) does 
not imply a considerable increase in computing time. However, computing 
the Bayes factor (and therefore the evidence) using the acceptance rate of 
the ABC algorithm is even faster. Moreover, it provides a different answer. 

Example — For the Poisson "P(#) model, if we take as an example an expo- 
nential £(1) prior 7T0, the evidence associated with the model is 

/nx -28 
— -de = 2- xo ~ 1 , 
x ! 



while the quantitative assessment of Ratmann et al. (20091 is 



vr e (A;|a;o)I{7r e (A;|xo) < 7r e (0|x )} , (1) 



-X'() 



with 



7r e (e|x ) OC / ; rr— kz&O 



(e + x )!(l + e 2 ) (1 + e 2 ) ' 

The numerical comparison of both functions of xq in Figure [T] shows a much 
slower decrease in xq for the p- value ((!]) than for the evidence, not to mention 
a frankly puzzling non-monotonicity of the p-value. < 



2 Implications of model criticism 



While the approach by Ratmann et al. (2009) provides an informal assess- 



ment that can be derived in an ABC setting, the Bayesian foundations of 
the method may be questioned. The core of the Bayesian approach is to in- 
corporate all aspects of uncertainty and all aspects of decision consequences 
into a single inferential machine that provides the "optimal" solution. In the 
current case, while the consequences of rejecting the current model are not 
discussed, they would most likely include the construction of another model. 
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Figure 1: Comparison of the decreasing rates of the evidence (blue) and of 



the p-value (black) derived from Ratmann et al. (20091 for a Poisson model 



In the first graph in the paper, several models are contrasted and this leads 
us to wonder about the gain compared with using the Bayes factor, which 
can be directly derived from the ABC simulation as well since the (accepted 
or rejected) proposed values are simulated from %(Q)f(x\9). 

Example — For the Poisson xo ~ 'Pifi) model, running ABC with no ap- 
proximation (since this is a finite setting) produces an exact evaluation of 
the evidence. < 

We also note that the non-parametric evaluation at the basis of the ABC^ 
algorithm of Ratmann et al. ( |2009 ) can equally be used for approximating 
the true marginal density m{x). The smooth version of ABC^ presented 
in Section SI. 5, eqn. [S8], is however far from being a density estimate of 
£,xo,e( e ) since it based on a single realisation from f(x\6). It should rather be 
construed as a (further) smoothed version of its smooth ABC counterpart 
and this suggests integreting over h as well. Unless some group structure 
can be exploited to avoid the repetition of simulations Xb = the non- 

parametric estimator [S9] cannot be used as a practical device because either 
B is small, in which case the non-parametric approximation is poor, or B 
is large, in which case producing the XbS for every value of 8 is too time- 
consuming. Obviously, using moderate B is always feasible from a compu- 
tational point of view and it can also be argued that the approximation of 
f p (9, e\xo) by f p (9, e\xo) is not of major interest, since the former is only an 
approximation to the true target. (In a vaguely connected way, the rejection 
sampler of Subsection SI. 8 does seem an approximation to exact rejection- 
sampling, in that the choice of the upper bound C = 
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over the samples simulated in Step 1 of the algorithm does not produce a 
true upper bound.) 



3 On the meaning of the error 

The error term e is defined as part of the model, based on the marginal, 



with the additional input of a prior distribution vr(e). Since Ratmann et al 



(20091 analyse this error based on the product of two densities, ^a; 0i e(e)7r(e), 
this product is not properly defined from a probabilistic point of view. The 
authors choose to call Cx ,e(^) a "likelihood" by a fiducial argument, but 
this is (strictly speaking) not [proportional to] a density in xq. Obviously, 
simulating from the density that is proportional to ^ XOt g(e)ir(e)TT(9) is entirely 
possible as long as this function integrates in [9, e) against the dominating 
measure, but it suffers from an undefined probabilistic background in that, 
for instance, it is not invariant under reparameterisation in e: changing e to e 
introduces the squared Jacobian \de/ds\ 2 in the "density". We acknowledge 
that most ABC strategies can be seen as using a formal "prior +likelihood" 
representation of the distribution of e, since 



vtabcW = J 7T e (e)£(e|x ,#)de^) 



but this formal perspective does not turn e into a "true" parameter and 
7r e into its prior. For instance, non parametric 7r e 's may be based on the 
observations or on additional simulations. 

The denomination of "likelihood" is thus debatable in that £ X0t g(e)7r(e)ir(9) 
cannot always be turned into a density on xq (or even on a statistic S(xo)). 

Example — For the Poisson xq ~ ~P(d) model, ^x ,e( e ) is t ne translated 
Poisson distribution V{9) — e, truncated to positive values. While this is 
indeed a distribution on xo, conditional on (9,e), it cannot be used as the 
original Poisson distribution, because of the unidentifiability of e. < 

We also think that comparing models via the ("posterior") distributions 
of the errors e does not provide a coherent setup in that this approach does 
not incorporate the model complexity penalisation that is at the heart of 
the Bayesian model comparison tools like the Bayes factor. First, a more 
complex (e.g., with more parameters) model will most likely have a more 
dispersed distribution on e. Second, returning to the first argument of that 
nore, the choice of the prior 7r(e) (and of the error e itself) is model dependent 
(as stressed in the paper via the notation 7r(e, M)) and the comparison thus 
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reflects possibly mostly the prior modelling instead of the data assessment, 
as shown, again, by the location parameter example. Using the same band 
of rejection for all models as in Figure 1 of Ratmann et al. (20091 thus does 
not seem possible nor recommendable on a general basis. 
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